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AUTOMATED TEST ITEM GENERATION SYSTEM AND METHOD 



CLAIM OF PRIORITY 
[0001] This application claims priority to U.S. provisional patent application no. 
60/461,896, filed April 10, 2003, entitled "Automated Test Item Generation System and 
Method," which is incorporated herein by reference in its entirety. 

FIELD OF THE INVENTION 
[0002] The present invention relates to methods and systems for generating assessment 
items automatically. More particularly, this invention relates to a computer system and method 
for providing an interface for automatically generating word problems for use in testing. 

BACKGROUND OF THE INVENTION 
[0003] The use of Automatic Item Generation (AIG), the practice of creating 
assessment items algorithmically, is of increasing interest to educational assessors. AIG permits 
educational assessors to quickly generate assessment items and at least partially automate the 
generation of such assessment items. Furthermore, assessors can provide differing assessment 
items, each having a similar difficulty level, to improve test security. Moreover, adaptive testing 
having assessment items that vary systematically in difficulty can be produced. Such advantages 
have encouraged the research and development of AIG technologies. 

[0004] However, AIG depends upon well-founded models of the cognitive abilities 
underlying performance. Where such models are lacking, AIG can only have heuristic 
usefulness. Conventional AIG research has been performed in areas where well-founded 
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cognitive theories support the development of AIG algorithms, such as matrix completion and 
analytical reasoning. Such areas are generally restricted in content and highly structured. 

[0005] In contrast, progress in AIG of verbal item types has been more limited, due to 
the openness of content and the considerable complexity of natural language. In open-ended 
verbal items, a strong preference exists for developing naturalistic items based upon actually 
published materials, and the most productive approaches have focused upon providing 
techniques to support test developers by supporting more efficient item selection and evaluation. 

[0006] Where constrained item types have required natural language generation, the 
treatment of verbal materials has been straightforward and generally uses verbal templates to 
generate items. Typical template-based natural language generation includes two salient 
properties: 1) a list of phrases or sentences with open slots; and 2) the random or pseudo-random 
insertion of words from predetermined lists into particular slots. Template-based generation has 
the advantage of being straightforward, quick and dependent upon existing items. However, AIG 
from such simple templates is clearly limited because natural language complexities cannot be 
captured within a template format. Moreover, since the strings manipulated by template-based 
systems have no theoretical status, they do not support any principled analysis of the language 
employed in any particular problem type. 

[0007] Conventional template-based AIG systems suffer from four distinct limitations: 
maintainability, output flexibility, output quality and an inability to easily produce multilingual 
outputs. In a template-based system, a large number of lists are stored and manipulated in order 
to generate textual output because each list is task or field specific. Accordingly, repetitive lists 
may be required to complete populate all task sets. 
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[0008] In addition, as the number of templates in a template-based system grows, it is 
more likely that the variety of templates disguises the systematic combination of a much smaller 
set of variables. 

[0009] Moreover, systems must resolve context-dependencies inherent in language, 
such as subject-verb agreement, selection restriction (i.e., one drives a car, but flies an airplane), 
definite-indefinite selection (i.e., a student or the student), and the like. Such dependencies are 
handled ad hoc in a template-based system. 

[0010] Finally, in order to produce a multilingual template-based system, a system 
maintainer must generate new templates for target language. Moreover, dependencies between 
templates and dependencies between entries in templates must be redefined for each template 
and/or combination of entries in the target language. As such, significant effort must be 
expended and significant resources must be dedicated to create a multilingual template-based 
system. 

[0011] What is needed is a method and system for improving conventional automatic 
item generation by using non-template-based algorithms for generating assessment item text. 

[0012] A need exists for a method and system for improving the maintenance of 
automatic item generation systems. 

[0013] A further need exists for an automatic item generation system and method that 
increases output variety. 

[0014] A still further need exists for an automatic item generation system and method 
that produces higher text quality. 

[0015] A further need exists for a method and system of automatic test generation that 
more easily permits multilingual textual output. 
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[0016] The present invention is directed to solving one or more of the problems 
described above. 



SUMMARY OF THE INVENTION 
[0017] A Natural Language Generation (NLG) system according to the present 
invention may be used to perform automatic item generation. The NLG system may use 
computational linguistic techniques to automate text generation. The NLG system may receive 
input and determine textual content from the input. The textual content may be abstractly 
formulated without reference to details at the level of sentences, phrases or words. One or more 
sentences may be planned based on the abstract content. Following this process, the exact 
wording for each sentence may then be formed. 

[0018] Thus, a major difference between an NLG system according to the present 
invention and a template-based system is that the specification of textual content is separated 
from the details of the wording. Building an NLG system entails constructing a model specifying 
possible content and separating it from a model specifying how that content is most naturally 
expressed. 

[0019] An NLG system of the present invention is more maintainable than a template- 
based system because a large number of lists need not be stored and manipulated in order to 
generate textual output. In an embodiment, the present invention separates knowledge about 
language in general from knowledge about specific tasks. Accordingly, the present invention is 
easier to maintain and modify to suit changing assessment needs and demands. 
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[0020] In addition, the output of the present invention may be more flexible than a 
template-based model. The present invention may be used to reduce variety between template- 
based lists in order to enable more flexible test generation. 

[0021] Moreover, systems must resolve context-dependencies inherent in language, 
such as subject-verb agreement, selection restriction (i.e., one drives a car, but flies an airplane), 
definite-indefinite selection (i.e., a student or the student), and the like. Unlike a template-based 
system, the present invention may include modules that resolve such dependencies automatically. 

[0022] Furthermore, the present invention may permit text generation in multiple 
languages more easily than a template-based system. In an embodiment, low-level language- 
specific details are segregated into language-specific modules, which permit transposition into 
high-level knowledge without requiring a complete system redesign. Instead, adapting an NLG 
system for a target language may merely require adjustment of the language-specific modules to 
describe a new set of linguistic, grammatical, and/or semantic rules and translation of the entries 
in the template. 

[0023] Considerable synergy between AIG within assessment theory and NLG within 
computational linguistics exists. Both AIG and NLG permit automatic generation of verbal 
material. Moreover, the verbal material is generated in complementary ways. While AIG 
focuses on controlling construct-relevant content, NLG focuses on lower-level document details. 

[0024] Another advantage of the present invention is the ability to distinguish between 
factors that significantly affect item difficulty, known as radicals, and factors that do not, known 
as incidentals. Conventionally, no a priori way of determining radicals and incidentals is known. 
In general, the determination of radicals and incidentals depends on the cognitive content of the 
assessment item and the task the assessment item requires respondents to perform. However, in 
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the present invention, the vast majority of variables are incidentals, such as the particular choice 
of words, grammatical constructions, and phrasing details. 

[0025] Verbal tasks may be separated into two types of elements: those which involve 
decoding (i.e., determining the meaning of text as written) and those which involve content 
manipulation (i.e., performing inference or other thought processes on the content after 
decoding). The present invention specifies how to encode what test subjects decode, so a direct 
relationship exists between the encoding of the present invention and decoding by a respondent. 
In other words, radicals generally involve content manipulation and wording typically interacts 
with radicals at a content level. 

[0026] The present invention is directed to a method and system of generating tasks that 
are drawn from a constrained universe of discourse, are susceptible to formalization, require little 
complex inference, and for which basic verbal comprehension (decoding/encoding) is not the 
tested subject matter. Such tasks may include, for example, mathematical word problems. Such 
mathematical word problems may include distance-rate-time problems, interest rate computation 
problems, taxation problems, production problems, physics problems, and the like. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0027] Aspects, features, benefits and advantages of the embodiments of the present 

invention will be apparent with regard to the following description, appended claims and 

accompanying drawings where: 

[0028] FIG. 1 depicts an exemplary high-level implementation of a system for 

performing natural language generation according to an embodiment of the present invention. 
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[0029] FIG. 2 depicts an exemplary screen shot of an existing word problem assessment 
item according to an embodiment of the present invention. 

[0030] FIG. 3 depicts an exemplary variable modification screen for a distance-rate- 
time assessment item according to an embodiment of the present invention. 

[0031] FIG. 4 depicts an exemplary detailed control screen for a distance-rate-time 
assessment item according to an embodiment of the present invention. 

[0032] FIG. 5 depicts an exemplary task-relevant problem structure for an assessment 
item according to an embodiment of the present invention. 

[0033] FIG. 6 depicts an exemplary measurement unit definition screen according to an 
embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0034] Before the present structures, systems and methods are described, it is to be 
understood that this invention is not limited to particular structures, systems, methodologies or 
protocols described, as these may vary. It is also to be understood that the terminology used in 
the description is for the purpose of describing the particular versions or embodiments only, and 
is not intended to limit the scope of the present invention. 

[0035] It must also be noted that as used herein, the singular forms "a," "an" and "the" 
include plural references unless the context clearly dictates otherwise. Thus, for example, 
reference to a "variable" is a reference to one or more variables and equivalents thereof known to 
those skilled in the art, and so forth. Unless defined otherwise, all technical and scientific terms 
used herein have the same meanings as commonly understood by one of ordinary skill in the art. 
Although any methods, devices and material similar or equivalent to those described herein can 
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be used in the practice of testing of embodiments of the present invention, the preferred methods, 
devices, and materials are now described. All publications mentioned herein are incorporated by 
reference. Nothing herein is to be construed as an admission that the invention is not entitled to 
antedate such disclosure by virtue of prior invention. 

[0036] The term "semantic frame," as used herein, refers to a method for organizing 
verbal content. In particular, a semantic frame makes it possible to analyze a particular type of 
word problem and isolate a series of variables with clear task-model relevance. 

[0037] Semantic frames may be organized as a hierarchy of frames. Each semantic 
frame may contain linguistic detail including the schematic structure and the vocabulary 
associated with the particular semantic frame. Semantic frames may be used to denote particular 
patterns of vocabulary for certain types of assessment items that are highly systematic and have 
constrained vocabulary and syntax, such as mathematical word problems. 

[0038] The term "schematic logical representation," as used herein, refers to 
information contained in a file pertaining to the syntax and vocabulary used to generate natural 
language for a given assessment item. A schematic logical representation may define one or 
more variables for which text may be generated. 

[0039] Based on the semantic frame that the schematic logical representation uses, the 
variables for an assessment item may be limited in their vocabulary (i.e., only a limited number 
of word choices may be available for each variable). For instance, in a Limousine variant of a 
distance-rate-time semantic frame, vehicle and driver variables may be constrained to be a 
limousine and a chauffeur, respectively. An exemplary schematic logical representation 
according to an embodiment of the present invention may represent the sentence "A car drove 
600 miles" logically as "VEHICLE|W TRAVEL(DISTANCE="600"|P)." In the schematic 
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logical representation, lexical elements (VEHICLE, TRAVEL, and DISTANCE) maybe 
represented as abstract labels, which are filled in by choosing a semantic frame and substituting 
appropriate lexical and grammatical materials. In addition, variables (|W and |P) and numeric 
values, such as the specification of distance, may permit the schematic logical representation to 
abstract itself from language-specific details of verbal expression while still providing a detailed 
specification of the content and the order in which it is to be presented. Each lexical element 
may represent a variable that ranges over a class of elements. The potential values for a lexical 
element may depend upon the selected semantic frame. 

[0040] The term "mental model," as used herein, refers to a structure with a limited 
number of component entities and relationships, subject to direct imagination and inspection. 
For example, assessment items typically vary along the following dimensions (exemplary values 
for distance-rate-time assessment items are provided): 

• The number of events (one or two, typically) 

• The semantic frame(s) associated with each event 

• The primary participants in each event (variations include operator v. passenger and the 
type of vehicle) 

• The identity of the primary participants (participants may be the same or different for 
each event and each participant may use the same vehicle or a different vehicle) 

• The secondary roles relevant to each event (e.g., distance, rate, time, route followed, start 
point, end point, start time and end time) 

• The identity of the secondary roles across events (i.e., specifying relationships among 
events) 
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[0041] FIG. 1 depicts an exemplary high-level implementation of a system for 
performing natural language generation according to an embodiment of the present invention. In 
the embodiment, three primary components are identified: the document structure generator 105, 
the logical schema generator 115, and the sentence generator 125. The document structure 
generator 105 receives as input a set of abstract choices from a user interface and produces, for 
example, an XML document 110. The XML document 110 may represent the underlying 
schematic content of an assessment item that is being generated. In an embodiment, the 
document structure generator 105 may perform, for example, the following tasks: building a 
mental model, outlining a document structure which expresses the mental model in a format 
compatible with the task-relevant problem structure, and passing choices about low-level 
wording to later modules. Building a mental model may include, for example, identifying which 
high-level semantic frames are to be used, setting up a list of events, and binding variables across 
events. Outlining a document structure may include, for example, determining the number of 
sentences required to express different parts of the mental model, determining the function of 
each sentence (e.g., stating a set of facts, querying a particular variable, and the like), and 
determining whether information is expressed in each sentence. Passing on the choices may 
include, for example, packaging the information for transmission to the logical schema 
generator 115. 

[0042] In an embodiment, the first intermediate representation 110 may be an XML 
document. The first intermediate representation 110 may specify the possible values for 
variables pertinent to the assessment item, indicate high-level items pertaining to the assessment 
item structure, and outline the sequence of sentences that will constitute the final assessment 
item. An exemplary first intermediate representation 110 is shown in Table 1. 
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<?xml><body> 

<variables>W,X,Y,Z,VONE</variables> 

<frame id="l" frameID="ONE" type="DRT" V0NE="VEHICLE7> 
<event id="2" frameID="ONE" eventID="A" type="DRT" 

subj="W@" rate="30" rateVar="Y" /> 
<event id="3" frameID="ONE" eventID="B" type="DRT" 

subj="X@" rate="70" rateVar="Z" /> 
<bindings id="4" frameID="ONE" type="DRT" W="VONE" X="VONE" /> 
<proposition id="2" frameID="ONE" eventED="A" role="QCColumnA" type="DRT" 

distance-"QUERY" rate-'EXPRESSED" time-"UNKNOWN" /> 
proposition id="F" frameID="ONE" eventED="B" role="QCColumnB" type="DRT" 

distance-"QUERY" rate- 'EXPRESSED" time-"UNKNOWN" /> 

</body> 

Table 1: An XML Word Problem Specification 

[0043] The information contained in Table 1 specifies that five variables are used (i.e., 
W, X, Y, Z, and VONE) and that the sentences in the resulting assessment item will use 
vocabulary from a single semantic frame of type DRT (distance-rate-time). The variable VONE 
is defined to be of type VEHICLE within the semantic frame. In addition, two events are 
described in the document, with different rate values and different subject variables. Moreover, 
the resulting document includes two sentences. Each sentence queries the DISTANCE variable 
and provides RATE information. Neither sentence defines any information regarding TIME. 
Furthermore, the first sentence is placed in column A of a word problem having a Quantitative 
Comparison format, and the second sentence is placed in column B under the same format. 

[0044] Although the content of the first intermediate representation 110 is abstract, the 
representation may indicate, in a concise form, the information that is to be presented and the 
organization of the information. The succeeding modules may define further details for the 
resulting assessment item. 

[0045] The logical schema generator 115 may perform, for example, two discrete tasks. 
First, the logical schema generator 115 may structure sentences in the resulting assessment item 
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so that the sentences are directly tied to the appropriate semantic frame. For example, for a 
distance-rate-time problem, the logical schema generator 115 may restructure the input from the 
first intermediate representation 110 so that a series of schematic statements or questions about 
VEHICLES, TRAVELing, RATE, TIME and DISTANCE are produced. 

[0046] In addition, the logical schema generator 115 may decide the format of the 
information contained in each sentence. Particularly, this may include the type of verb to use and 
the ordering of elements. The logical schema generator 115 may produce a predicate calculus in 
which the arguments are specified by the underlying mental model and in which the predicates 
are variables to be filled by actual natural language expressions. However, the logical schema 
generator 115 may not engage in fine details regarding the phrasing of each sentence. Instead, 
the transformation of the first intermediate representation 110 to the second intermediate 
representation 120 may be based on detailed information about the semantic frames relevant to 
each problem type. In an embodiment, the language for.the assessment item is stored in the 
second intermediate representation 120. The second intermediate representation 120 may result 
in an output divided into two sections. The first section may include a series of variable 
definitions, which identify the classes of vocabulary to be used whenever a particular variable 
occurs in the text. The second section may include a logical formula using these variables, which 
sets up a schema or template for the resulting assessment item. 

[0047] In an embodiment, the second intermediate representation 120 may be an XML 
document. The XML document may specify variable definitions including a series of tags for 
indicating how particular logical expressions map to actual vocabulary. Exemplary mappings are 
shown in Table 2. 
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<frameset id="DRT" type="9998" /> 
<frameset id="TEMP" type=" 10002" /> 

<vardef name="DRTVE" id="DRT" frameRole="27" /> 
<vardef name="VEHICLE" id="DRT" frameRole="3" /> 

<vardef name="TIME" id="TEMP" frameRole="2" unit="5" /> 

Table 2: Logical Expression Mappings 

[0048] The definitions in Table 2 may define two semantic frames by reference to each 

semantic frame's location in a database containing the equivalent of a dictionary specification of 

all semantic frame elements. The variable DRIVE may be defined to use vocabulary from role 

27 in the DRT semantic frame definition, the variable VEHICLE may be defined to use 

vocabulary from role 3 in the DRT semantic frame definition, and the variable TIME may be 

defined to use vocabulary from role 2 in the TEMP semantic frame definition. Based on these 

and other variable definitions, the remainder of the output may include a precise schematic 

specification of the content to be output. Table 3 shows a specification instance. 



<given> 

((W and X each)|SB :: PERSON|SB 

DRTVE(SEPARATE : VEHICLE|OB, SAME : DISTANCE|DI, SAME : NUMBER : TEV1E|TI) 

</given> 

<given> 

PERSON|W LEAVEONE(ORIGIN|I, CLOCKTIME|M) and PERSON|W 
ARRTVEONE(DESTINATION|K, CLOCKTDVIE|0), 

DRTVEONE((VONE|Y :: VEHICLE|Y), (METER="50"|S :: DISTANCE^), STRAIGHT : 

ROUTE|G) 
</given> 
<columnA> 

AVERAGE : NUMBER : RATE :: PERSON|X 

DRTVEONE((VONE|Z :: VEHICLE|Z), (TI|U :: TEVIE|U), EVENT|F :: PERSON 
TRAVELONE|EVENT((DI|S :: DISTANCE^), STRAIGHT : ROUTE|H, ORIGIN|J, _ 

DESTINATION|L)) RANGE(START|N, END|P) 
</columnA> 
<columnB> 
RATE="3" 

</columnB> 

TABLE 3: Logical Schema Representation 
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[0049] The representation in Table 3 varies from standard logical representations in 
order to make the translation to actual natural language text more straightforward. The 
translation is made more straightforward by, for example, separating the subject from the 
predicate; treating variables, such as |N, |S and |P, as labels attached to the elements that are 
verbally expressed; by treating modifiers as being attached to the argument that they modify; and 
the like. 

[0050] The sentence generator 125 may translate the logical representations in the 
second intermediate representation 120 into assessment item text. The sentence generator 125 
may parse the logical representations, annotate the resulting parse tree with grammatical 
information, find words and word forms in a dictionary based related to a selected language to fit 
the grammatical information and output the resulting text in order to complete the translation. 

[0051] The sentence generator 125 may cover a wide range of linguistic details that are 
generally unrelated to the substance of the assessment item. For example, the sentence generator 
125 may record the number of times an entity has been mentioned so that the choice of 
determiners between "a"/"an" and "the" is appropriate. Other grammatical phenomena may also 
be tracked, such as subject/verb agreement, preposition and case marking, and the like. 

[0052] The output representation 130 may be an XML document having all text 
converted to the specified language. The output representation 130 may retain XML tags 
identifying the role that each text chunk performs in the structure of the assessment item. The 
output representation 130 may then be formatted for display and presented as an assessment item. 
The resulting assessment item may be used in connection with a test creation system. 
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[0053] FIGs. 2-6 depict exemplary screen shots of a graphical user interface used to 
generate assessment items according to an embodiment of the present invention. In the 
embodiment disclosed in FIGs. 2-6, a distance-rate-time assessment item is generated. Other 
embodiments include interest accumulation problems, tax problems, problems associated with 
producing items and the like. The variables used in each embodiment may be selected based on 
the problem type to be solved. Accordingly, the embodiment described below and in reference to 
FIGs. 2-6 is not meant to be limiting, but merely exemplary of the generation of one type of an 
assessment item. 

[0054] Generation of the distance-rate-time assessment item may be sub-divided into 
five tasks: assigning mental model structure variables, defining identity variables in the mental 
model structure, determining a task-relevant problem structure, describing a document format 
and structure, and determining language variations. 

[0055] The assignment of mental model structure variables may include defining the 
number of events; the number of distinct frames; the types of participants; the type of trip; the 
default unit of measure; and the actual distance, rate and time units used in each event. In an 
embodiment, a distance-rate-time problem has one or two events. Alternately, more events may 
be used for a distance-rate-time problem. If two or more events are used, the semantic frame for 
each event may differ. For example, a speed of a truck may be at issue in a first event, and a 
speed of a train may be at issue in a second event. Alternately, the semantic frames for each 
event may be the same. The type of participant may be used to specify whether people and/or 
vehicles are mentioned in the assessment item. If more than one event is included in the 
assessment item, the primary participants may include different people using the same vehicle or 
different vehicles, or the same person using the same vehicle or different vehicles. The trip type 
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may be used to determine the values of one or more secondary variables. The trip type may 
include a round trip, a one way trip, two people or vehicles heading towards each other, two 
people or vehicles heading away from each other, and numerous other types of trips. 

[0056] Defining the identity variables in the mental model structure may include 
choosing variables to express one or more of the following in each event: people, vehicles, time, 
distance, rate, starting locations, ending locations, starting times, ending times, the route and the 
event as a whole. While the distance, rate and time values are central to determination of the 
problem, other variables may be purely incidental to the determination of the answer to the 
problem. 

[0057] Determination of a task-relevant problem structure may include determining 
which variable is being determined for each event, the answer for each event, numeric values for 
the mathematical variables (i.e., numeric values for distance, rate and time in each event), 
whether some information is provided in summary form, and whether time is expressed as a 
sequence of clock times. Most problems within a Transportation semantic frame may solve the 
equation d=rt. In such a case, the equation may map to the semantic roles Distance, Rate and 
Time. Another problem type that may be within a Transportation semantic frame may include a 
fuel efficiency problem. In such a problem, the equation to solve may be f=ed, where f is fuel 
consumption, e is fuel efficiency, and d is distance. If a unique answer is required for solution to 
an assessment item, only one variable may be unspecified. Accordingly, values may be assigned 
to all but one variable to provide for a unique solution. Implicit values may be assigned to 
complicate an assessment item. For example, the amount of time may not be explicitly specified. 
Rather, the assessment item may state that the person/vehicle traveled from 3:00 to 6:00. 
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[0058] Describing a document format and structure may include determining the format 
of the problem (i.e., quantitative comparison vs. problem solving) and the arrangement of content 
between the options and the stem. The problem format may determine the type of language used 
for the query of the assessment item. For example, a problem-solving (multiple choice) 
assessment item may query information in the form of a question: "Which of the following?" 
"How much?" or "How many?" In contrast, a quantitative comparison question may query 
information in a phrasal form: "The number of hours required. . .," "The time taken to. . .," "The 
speed at which. . ." Moreover, content may be arranged in different sections of an assessment 
item. For instance, in a quantitative comparison assessment item, both background information 
and the questions may be posed in the columns. Alternatively, background information may be 
posed in a question stem and the questions may be posed in the columns. Such arrangements are 
merely exemplary. Other embodiments may arrange information in alternate arrangements 
without limitation. 

[0059] Determining language variations may include, for example, selecting a language, 
selecting a sentence structure, selecting referent identification types (identifying people by name 
or generically, identifying object with labels or by description), selecting detailed phrasing, 
determining whether quantities are in integer format, determining whether rates are identified as 
constant or average, and whether events are described in the present or past tense. The language 
may be selected based on a variable assigned by a user. In an embodiment, different data 
structures are stored for each language. The data structures may include, without limitation, 
words (in the selected language) available for use in assessment items and rules pertaining to 
sentence structure and grammar. The sentence structure may include an active or passive verb 
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form. Referent identification may include calling a person in an event, for example, "Fred," "a 
driver," "the driver," or "driver A." 

[0060] FIG. 2 depicts an exemplary screen shot of an existing word problem assessment 
item according to an embodiment of the present invention. The menu bar may be used to modify 
one or more variables for the assessment item. 

[0061] FIG. 3 depicts an exemplary variable modification screen for a distance-rate- 
time assessment item according to an embodiment of the present invention. In an embodiment, 
the variable modification screen permits a user to alter the variable for which the assessment item 
solves 305, the basic document format 310, the core participant type 315, and the number of 
events 320. 

[0062] FIG. 4 depicts an exemplary detailed control screen for a distance-rate- time 
assessment item according to an embodiment of the present invention. In an embodiment, the 
detailed control screen permits a user to alter the number of distinct vehicle types 405, the name 
used for each person 410, the type of trip 415, and numerous other parameters pertaining to the 
syntactic structure and the secondary grammatical content. 

[0063] FIG. 5 depicts an exemplary task-relevant problem structure for an assessment 
item according to an embodiment of the present invention. In an embodiment, at least one of the 
distance, rate and time variables 505 and 510 maybe set for each event in the assessment item. 
If desired, the user may assign constant values 515 to the assessment item. In addition, an 
answer 520 to the assessment item may be supplied if desired. If an answer is supplied, values 
may be selected from the ranges defined for each event to ensure that the specified answer is 
correct for the resulting assessment item. If any value is unspecified, a random value may be 
selected for the variable. 
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[0064] FIG. 6 depicts an exemplary measurement unit definition screen according to an 
embodiment of the present invention. Default units may be selected for one or more of a rate, a 
time and a distance either globally 605 or on a per event basis 610 and 615. 

[0065] Using the graphical user interface described in FIGs. 2-6, new assessment item 
instances may be generated from an underlying model Alternatively, parameters may be altered 
in an existing assessment item to generate a new assessment item. 

[0066] Although this invention has been illustrated by reference to specific 
embodiments, it will be apparent to those skilled in the art that various changes and 
modifications may be made which clearly fall within the scope of the invention. The invention is 
intended to be protected broadly within the spirit and scope of the appended claims. 
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